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METHODS FOR ESTIMATING THE POSITION AND SHAPE OF LIPS AND 
FOR ESTIMATING THE POSITION OF TEETH IN A SEQUENCE OF 
DIGITAL IMAGES OF A HUMAN FACE 

BACKGROUND OF THE INVENTION 

[0001] A sequence of digital images, such as, for example, a sequence of video 
frames of a talking person, may include images of a human face. In various 
computerized applications, it may be desirable to estimate features of the mouth, such 
as, for example, the location and shape of the lips in images of that sequence. 
Deformable mouth model templates may be used for that purpose. 
[0002] A deformable mouth model template is a model approximating the location 
and shape of a mouth. For the deformable mouth model template to estimate the 
shape and location of the mouth for a given image of a human face, an energy 
objective function may be minimized. 

[0003] The quality, accuracy and robustness of the estimation of mouth features may 
depend on the structure of the deformable mouth model template used, and/or on the 
energy objective function used, and/or on the minimization algorithm used. 
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BRIEF DESCRIPTION OF THE DRAWINGS 

[0004] Embodiments of the invention are illustrated by way of example and not 
limitation in the figures of the accompanying drawings, in which like reference 
numerals indicate corresponding, analogous or similar elements, and in which: 
[0005] FIG. 1 is an illustration of a deformable mouth model template, according to 
some embodiments of the invention; 

[0006] FIG. 2 is a flowchart of an exemplary method for estimating the position and 
shape of lips and for optionally estimating the position of teeth in one of a sequence of 
digital images of a human face, according to some embodiments of the invention; 
[0007] FIG. 3 is a flowchart illustration of an exemplary method for initializing a 
deformable mouth model template from a mouth image in a base image, according to 
some embodiments of the. invention; 

[0008] FIG. 4 is a flowchart illustration of an exemplary method for deriving a 
deformable mouth model template of a mouth in one of a sequence of digital images 
of a human face, according to some embodiments of the invention; 
[0009] FIG. 5 is an illustration of an exemplary second-level lip transformation, 
according to some embodiments of the invention; 

[0010] FIG. 6 is an illustration of an exemplary first-level lip transformation, 
according to some embodiments of the invention; 

[0011] FIG. 7 is an illustration of an exemplary zero-level lip transformation, 
according to some embodiments of the invention; 

[0012] FIGS. 8 - 10 are a flowchart illustration of an exemplary method for 
estimating the position and shape of lips in one of a sequence of digital images of a 
human face, according to some embodiments of the invention; 

[0013] FIG. 11 is an illustration of an exemplary first-level teeth transformation, 
according to some embodiments of the invention; 

[0014] FIG. 12 is an illustration of an exemplary zero-level teeth transformation, 
according to some embodiments of the invention; and 

[0015] FIGS. 13 — 14 are a flowchart illustration of an exemplary method for 
estimating the position of the teeth in one of a sequence of digital images of a human 
face, according to some embodiments of the invention; and 
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[0016] FIG. 15 is a simplified block diagram of an exemplary apparatus to execute a 
method for estimating position and shape of lips and for optionally estimating position 
of teeth in one of a sequence of digital images of a human face according to some 
embodiments of the invention. 

[0017] It will be appreciated that for simplicity and clarity of illustration, elements 
shown in the figures have not necessarily been drawn to scale. For example, the 
dimensions of some of the elements may be exaggerated relative to other elements for 
clarity. 
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DETAILED DESCRIPTION OF EMBODIMENTS OF THE INVENTION 

[0018] In the following detailed description, numerous specific details are set forth in 
order to provide a thorough understanding of embodiments of the invention. However 
it will be understood by those of ordinary skill in the art that the embodiments of the 
invention may be practiced without these specific details. In other instances, well- 
known methods, procedures, components and circuits have not been described in 
detail so as not to obscure the embodiments of the invention. 

[0019] FIG. 1 is an illustration of a deformable mouth model template 2 of a mouth, 
according to some embodiments of the invention. 

[0020] Deforniable mouth model template 2 may comprise lip contours 4, 6, 12 and 
14, and may optionally comprise teeth contours 8 and 10. In addition, deformable 
mouth model template 2 may comprise a left corner control point 16 (denoted 
"point Jeft" (P_L)) and a right corner control point 18 (denoted "point_right" (P_R)) 
that represent the left and right corners of the mouth, respectively. 
[0021] The following table lists the contours, what they represent, and how they are 
denoted hereinbelow. 



Table 1 



Contour 


Represents 


Denoted 


4 


upper boundary of upper lip 


contour_up_up (C_UU) 


6 


lower boundary of upper lip 


contour_up_down (C_UD) 


8 


lower boundary of upper teeth 


contour_teeth_up (C_TU) 


10 


upper boundary of lower teeth 


contour_teeth_down (C_TD) 


12 


upper boundary of the lower lip 


contour_down_up (C_DU) 


14 


lower boundary of the lower lip 


contour_down_down (C_DD) 



[0022] It is assumed that the upper boundary of the upper teeth coincides with the 
lower boundary of the upper lip, and that the lower boundary of the lower teeth 
coincides with the upper boundary of the lower lip. 

[0023] Control points 16 and 18 are common points of contours 4, 6, 8, 10, 12 and 14. 

[0024] Deformable mouth model template 2 may comprise control points 20, 22, 28 

and 30, located substantially on the centers of Up contours 4, 6, 12 and 14, 

respectively. In addition, deformable mouth model template 2 may optionally 
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comprise control points 24 and 26, located substantially on the centers of teeth 
contours 8 and 10, respectively. The locations of control points 20, 22, 24, 26, 28 and 
30 substantially on the centers of the respective contours are denoted "point_center" 
(PC). 

[0025] Deformable mouth model template 2 may comprise control points 32, 34, 40 
and 42, located on lip contours 4, 6, 12 and 14, respectively, between control point 18 
and the respective P_C. In addition, deformable mouth model template 2 may 
comprise control points 36 and 38, located on teeth contours 8 and 10, respectively, 
between control point 1 8 and the respective P_C. Although the present invention is 
not limited in this respect, control points 32, 34, 36, 38, 40 and 42 may be located on 
contours 4, 6, 8, 10, 12 and 14, respectively, substantially quarter of the length of the 
respective contour from control point 18. The locations of control points 32, 34, 36, 
38, 40 and 42 on the respective contours are denoted "pointrightcenter" (P_RC). 
[0026] Similarly, deformable mouth model template 2 may comprise control points 
44, 46, 52 and 54 located on lip contoxirs 4, 6, 12 and 14, respectively, between 
control point 16 and the respective P_C. In addition, deformable mouth model 
template 2 may comprise control points 48 and 50, located on teeth contours 8 and 10, 
respectively, between control point 16 and the respective P_C. Although the present 
invention is not limited in this respect, control points 44, 46, 48, 50, 52 and 54 may be 
located on contours 4, 6, 8, 10, 12 and 14, respectively, substantially quarter of the 
length of the respective contour from point 16. The locations of control points 44, 46, 
48, 50, 52 and 54 on the respective contours are denoted 6t point_left_center" (P_LC). 
[0027] The following table lists the control points of the mouth model template, what 
they represent, and how they are denoted hereinbelow. 



Table 2 



Control Point 


Represents 


Denoted 


16 


left corner of mouth 


P_L 


18 


right corner of mouth 


P_R 


20, 22, 24, 26, 28 and 30 


point at center of contour 


PC 


32, 34, 36,38, 40 and 42 


point on contour between P_C and P_R 


P_RC 


44, 46, 48, 50, 52 and 54 


point on contour between P_C and P_L 


PJLC 
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[0028] The shapes of the contours between control points may be calculated using 
conventional cubic splines. The Y coordinates of all the control points and the X 
coordinates of P L and P_R are freely determinable, while the X coordinates of the 
P_LC, P_C and P_RC control points are determined based upon their relative position 
to mouth corners (V4, Vi, and 3 A). 

[0029] The following table lists areas of the mouth model template, and how they are 
denoted hereinbelow. 



Table 3 



Area Denoted 


Meaning 


AUL 


upper lip (area enclosed by lip contours CMJU and C_JJD) 


AuT 


upper teeth (area enclosed by contours C_UD and C_TU) 


Amc 


mouth cavity (area enclosed by teeth contours C_TU and C TD) 


Adt 


lower teeth (area enclosed by contours C TD and C_JDU) 


Adl 


lower lip (area enclosed by lip contours CJDU and C_DD) 


Alc 


left corner of mouth (area surrounding control point PJL) 


Arc 


right comer of mouth (area surrounding control point P_R) 



[0030] Alternatively, the area enclosed by Up contours C_UD and C_DU may be 
denoted "AjjT+MC+DT" 

[0031] FIG. 2 is a flowchart illustration of an exemplary method for processing a 
sequence of digital images of a human face, according to some embodiments of the 
invention. The method of FIG. 2 may be implemented in any of a variety of 
applications, such as, for example, computer vision, image understanding, object 
detection, human-machine interfaces, automated visual surveillance, face 
detection/recognition, communication, lip synching, synthetic video, education and 
games. 

[0032] Processing the sequence of digital images may begin with processing of a 

single digital image of the sequence, referred to as "the base image" (70). If there is 

an additional digital image to be processed (72), then the additional digital image is 

processed (74). If not, then processing of the sequence is terminated. 

[0033] Processing the base image may include a method of initialing a deformable 

mouth model template based on the image of a mouth in the base image. An example 
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of such a method is described hereinbelow with respect to FIG. 3. Processing the 
additional image may include a method of estimating the position and shape of lips 
and optionally the position of teeth of a mouth in the additional image. An example 
of such a method is described hereinbelow with respect to FIG. 4. 
[0034] FIG. 3 is a flowchart illustration of an exemplary method for initializing a 
deforrnable mouth model template from a mouth image in a base image according to 
some embodiments of the invention. For example, a deforrnable mouth model 
template is initialized once pixel coordinate values have been given to the contours 
and control points of the template. 

[0035] The mouth area in the base image may be copied into a new image — denoted 
mouth image — having a fixed size, for example, a width 46 W" of 128 pixels and a 
height "H" of 128 pixels. The mouth area in the base image may be scaled and 
rotated to fit to the size W*H of mouth image Jjj. A mouth in a neutral position, e.g. 
not stretched and not squeezed, may have a width of, for example, W/2 in mouth 
image I B (80). 

[0036] Lip contours 4, 6, 12 and 14 and control points 16, 18, 20, 22, 28, 30, 32, 34, 

40, 42, 44, 46, 52 and 54 of a deforrnable mouth model template Mg may be 

initialized by a known or future, automatic or manual initialization method (82). 
[0037] If an estimation of the position of the teeth in the sequence of images is not 
required (84), the processing of the base image is terminated. Otherwise, if the teeth 
are hidden (88), teeth contours 8 and 10 may be initialized to coincide with lip 
contours 6 and 12, respectively, and control points 36, 24 and 48, and 38, 26 and 50 
may be initialized to coincide with control points 34, 22 and 46, and 40, 28 and 54 
respectively (90). 

[0038] However, if the teeth are not hidden, teeth contours 8 and 10 and control 

points 24, 26, 36, 38, 48 and 50 of the deforrnable mouth model template Mb may be 

initialized by a known or future, automatic or manual initialization method (94). 
[0039] In the following description, an image being processed in (74) is referred to as 
a "current" image. In addition, during the processing of a current image, information 
from a second image may be used, such as for example, the estimated position and 
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shape of the lips and the estimated position of the teeth of the second image. In the 
following description, that second image is referred to as a "previously processed" 
image. 

[0040] However, the term "previously processed" is not intended to imply a temporal, 
sequential or spatial relationship between the two images. It is possible that the 
previously processed image occurs earlier in the sequence than the current image. It 
is also possible that the previously processed image occurs later in the sequence than 
the current image. Moreover, it is possible that the previously processed image and 
the current image are not consecutive in the sequence. 

[0041] FIG. 4 is a flowchart illustration of an exemplary method for deriving a 
deformable mouth model template of a mouth in one of a sequence of images of a 
human face, according to some embodiments of the invention. 

[0042] The mouth area may be copied into a new image -denoted mouth image "i" - 
having a fixed width W and a fixed height H. The mouth area may be scaled and 
rotated to fit the size W*H of mouth image "7" (102). An image intensity map I(x, y) 
may be calculated by known or future methods (104), and a spatial luminance peaks 

and valleys map f EAK VALLEY (x,y) may be calculated by known or future methods 

(106). A teeth vertical gaps map F—(x,y) may be calculated (108), according to the 
following equations: 



where in equation (2), "d" is an image difference step and may have a value of several 
pixels, for example, 2, and ItHRESHOLD ma Y be an empirical threshold for the teeth 

area, with a value, for example, of approximately 30 (on the assumption that pixel 
intensity values range from 0 to 255), and the function "1" is defined in equation (3) 
as having the value zero when its argument is less than or equal to zero and as having 
the value one otherwise. 



(1) 



(2) 



I TG (x,y) = l(v).V 

V = (l(x-d,y)+l(x + d,y)- 2l(x, y) - J, 



THRESHOLD 




8 



Attorney Docket No.: P-62 1 4-US 
Assignee: Intel Corporation 

V-EDGE 

[0043] A vertical edges map / (x,y) of the vertical intensity gradient may be 

used in estimating the position and shape of the lips in an image of a sequence of 

/EDGE 
fay) may be calculated by 

known or future methods (110), for example, as described in the following equation: 

/(*,y + l)-J(s,y-l) 
2 

[0044] The position and shape of the lips are estimated (1 12) using, for example, the 
methods shown hereinbelow in FIGS. 8, 9 and 10. Optionally, the position of the 
teeth is estimated (1 14) using, for example, the methods shown hereinbelow in FIGS. 
11 and 12. 

ESTIMATING THE POSITION AND SHAPE OF THE LIPS 

[0045] The position and shape of the lips are estimated in an iterative process using 

three different types of transformations. A "second-level lip transformation" T2i(DX, 

DY) is a coarse transformation used once to transform the location of a deformable 
mouth model template of a previously processed image to a location generally 
reflecting the location of the mouth in the current image. Briefly, FIG. 5 illustrates an 
exemplary second-level lip transformation on a deformable mouth model template, 
with contours C_TU and C_TD omitted for clarity. Template 500 is the template 
prior to the transformation, and template 502 is the template after the transformation. 
DX and DY are relative displacements of the center of the deformable mouth model 
template along perpendicular axes X and Y, respectively, as summarized below in 
Table 4. In the second-level lip transformation, the locations of the control points of 
the lip contours C_UU, C UD, C DU and C_DD are transformed in unison, as shown 
in Table 5, which is explained in detail hereinbelow. 

[0046] A "first-level lip transformation" T IL (DX, DY, DYu, DYd, DW) is a fine 

transformation used to refine the estimated location of the mouth and to generally 
estimate the opening and stretching of the mouth. Briefly, FIG. 6 illustrates an 
exemplary first-level lip transformation on a deformable mouth model template, with 
contours C_TU and C_TD omitted for clarity. Template 600 is the template prior to 
the transformation, and template 602 is the template after the transformation. As 
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before, DX and DY are relative displacements of the center of the deformable mouth 
model template along perpendicular axes X and Y, respectively. A dashed template 
illustrates the displacement from template 600. DW is a difference in the mouth 
stretching, DYu is the difference in the location of the upper lip relative to the center 
of the mouth, and DYd is the difference in the location of the lower lip relative to the 
center of the mouth, as summarized below in Table 4. In the first-level lip 
transformation, the locations of the control points of contours C_UU and C_UD are 
transformed in unison, and the locations of the control points of contours C_DU and 
C_DD are transformed in unison, as shown in Table 5, which is explained in detail 
hereinbelow. 

[0047] A "zero-level lip transformation" T 0L (XL, XR, YL 3 YR, YDISPL(i,k)) is a 

superfine transformation used to refine the estimated location of individual lip control 
points of the deformable mouth model template. Briefly, FIG. 7 illustrates an 
exemplary zero-level lip transformation on a deformable mouth model template, with 
contours C_TU and C_TD omitted for clarity. Template 700 is the template prior to 
the transformation, and template 702 is the template after the transformation 
(displaced relative to template 700 only for clarity). XL and YL describe Hie relative 
horizontal and vertical displacements, respectively, of control point P_L. Similarly, 
XR and YR describe the relative horizontal and vertical displacements, respectively, of 
control point P R. YDISPL(i t k) describe the vertical displacements of each of the lip 
control points 20, 22, 28, 30, 32, 34, 40, 42, 44, 46, 52 and 54. These transformation 
parameters are summarized below in Table 4. In the zero-level lip transformation, the 
locations of the control points of contours CMJU, C_UD, C_DU and C_DD are 
individually transformed, as shown in Table 5, which is explained in detail 
hereinbelow. 

[0048] The first-level and zero-level lip transformations may be used several times in 

a defined order to refine the estimation of the position and shape of the lips. 

[0049] It should be understood that in alternative embodiments of the invention, a 

different number of lip transformation types may be used. Moreover, it should be 

understood that in alternative embodiments of the invention, lip transformation types 

may be used a different number of times and in a different order or in no specific 

order. 
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[0050] FIGS, 8, 9 and 10 are a flowchart illustration of an exemplary method for 
estimating the position and shape of lips in one of a sequence of images of a human 
face, according to some embodiments of the invention. 

[0051] Referring to FIG- 8, a deformable mouth model template M for the current 
image is initialized to be equal to a deformable mouth model template Mp of a 
previously processed image (202). 

[0052] Mouth image 7/? of the base image, mouth image Ip of the previously 

processed image, and mouth image / of the current image are blurred with a W/32 
window using known or future methods to generate a "double-blurred mouth image 

a "double-blurred mouth image 7p", and a "double-blurred mouth image F\ 

respectively (204). 

[0053] An exemplary blurring method to be used in (204) is presented in the 
following equation: 

w/2 w/2 



1 w/2 w/2 

(6) 2?[/(x,;/), W ] = _i- £ ^l(x + Uy + j) 



where w is the size of the window. However, mouth images J#, Ip 3 and / need not be 

blurred by the same method, nor need they be blurred in any particular order. 

[0054] "Double-blurring" using a window size of W/32 of an exemplary mouth image 

I(x,y) using equation (6) is demonstrated in the following equation: 



T( \ W 



(7) Double - Blurred I(x, y) = B 
V EDGE 

[0055] Vertical edges map / fay) and the spatial luminance peaks and valleys 

map jf^^ VALLEY fay) are filtered using known or future methods, for example, a 

11 MAX" filter (also known as a "dilate morphological operator") with a window, 
to generate a "double-filtered vertical edges map" and a "double-filtered spatial 
luminance peaks and valleys map", respectively (210). 

[0056] An exemplary "MAX" filtering method to be used in (210) is presented in the 
following equation: 



11 



Attorney Docket No.: P-6214-US 
Assignee: Intel Corporation 

m MAX[l(x,y),w]= max l(x + i,y + j) 

2 2 2 2 

where w is tlie size of the window. However, maps f (x,y) and f 



EAK- 



VALLEY 

(x t y) need not be filtered by the same method, nor need they be filtered in any 
particular order. 

[0057] "Double-filtering" using a window size of WI32 of an exemplary mouth 
image I(x,y) using equation (8) is demonstrated in the following equation: 

(9) Double - Filtered I(x,y) = MAx\l(x,y\— 

L 32 _ 

[0058] Transformation parameters DX and DY for a second-level lip transformation 
T2i(DX, DY) are determined by mmimizing a lips energy objective function E^, 
which is defined in equation (21) and calculated using double-blurred mouth images 
Ifi, Ip, and /, the double-filtered spatial luminance peaks and valleys map and the 
double-filtered vertical edges map (214). 

[0059] Determining parameters DXand DYis demonstrated in the following equation 
and may be achieved using known or future methods, such as, for example, gradient 
descent, downhill-simplex, and the like. 

(10 ) DX,DY = KgmmE L (T 2L {DX,DY),l) 

[0060] A deformable mouth model template Mj is generated (216) by transforming 

deformable mouth model template M using T2l(DX, DY), and a variable denoted 

"initO", serving as a loop counter, is initialized to the value zero (218). 

[0061] If variable initO is equal to zero (220), deformable mouth model template M is 

initialized with deformable mouth model template Mj (222), else, deformable mouth 

model template Mis initialized with deformable mouth model template Mb (224). 

[0062] Mouth image 1% of the base image, mouth image Ip of the previously 
processed image, and mouth image / of the current image are blurred with a WI64 
window using, for example, equation (6), to generate a "blurred mouth image Jg", a 
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"blurred mouth image Ip", and a "blurred mouth image 7", respectively (226), as 
shown in the following equations: 



(11) Blurred I B (x,y) = B I B { x ,y\^ 

(12) Blurred I p (x, y)= B I p ( x ,y),^ 

(13) Blurred = B '(*,J^] 



V EDGE 

[0063] Referring now to FIG. 9, vertical edges map I " (x,y) and the spatial 
luminance peaks and valleys map j PEAK VALLEY (x,y) are filtered using known or 

future methods, for example, the "MAX" filter presented in equation (8) with a WI6A 
window, to generate a "filtered vertical edges map" and a "filtered spatial luminance 
peaks and valleys map", respectively (232). 

[0064] Transformation parameters DX, DY, DYu, DYd and DW of first-level lip 
transformation T 1L (DX, DY t DYu, DYd, DW) are determined by minimizing lips 

energy objective function E^, which is defined in equation (21) and calculated using 

blurred mouth images /g, Ip, and I, the filtered spatial luminance peaks and valleys 

map and the filtered vertical edges map (236). Determining parameters DX, DY, DYu, 
DYd and DW is demonstrated in the following equation and can be obtained using 
known or future methods, such as, for example, gradient descent, downhill-simplex, 
and the like. 

(\4)DX,DY,DYit,DYd,DW = arg min E L (T„(DX,DY,DYu,DYd 9 DW\l) 

[0065] A deformable mouth model template M2 for the current image is generated 
(238) by transforming deformable mouth model template M using first-level lip 
transformation T IL (DX, DY, DYu, DYd, DW). 

[0066] A variable denoted "iraYi", serving as a loop counter, is initialized to the value 
zero (240). Deformable mouth model template M is initialized with deformable 

mouth model template M2 (242). 
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[0067] If variable initl equals 1 (244), control point 16 of deformable mouth model 
template M is shifted left on the "X" axis by Will (246). If variable initl equals 2 
(248), control point 1 8 of deformable mouth model template M is shifted right on the 
"X" axis by 0712 (250). 

[0068] Transformation parameters XL, XR, YL, YR and YDISPL(i,k) of a zero-level lip 
transformation T 0L (XL, XR, YL YR, YDISPL(i,k)) are determined by minimizing lips 

energy objective function P^, defined in equation (21) and calculated using mouth 

images 7g s Ip, and 7, the spatial luminance peaks and valleys map and vertical edges 
map (252). 

[0069] Detemiining parameters XL, XR, YL, YR and YDISPL(i,k) is demonstrated in 
the following equation and can be obtained using known or future methods, such as, 
for example, gradient descent, downhill-simplex, and the like. 
XL,XR,YL,YR,YDISPL(i,k) = 
(15) arg min E L (T QL (XL, XR, YL, YR, YDISPL(i, k)\ /) 

[0070] Referring now to FIG. 10, a deformable mouth model template M3 for the 
current image is generated (254) by transforming deformable mouth model template 
Musing zero-level lip transformation T 0L (XL, XR, YL, YR, YDISPL(i,k)). 

[0071] Energy E3, corresponding to deformable mouth model template M3, is 
calculated using contour and control point values of deformable mouth model 
template M3 hi the equation for lips energy objective function (256). 

[0072] The minimum energy of hps energy objective function P^ once the lips 

estimation procedure is completed may be denoted E*. In addition, the final 
deformable mouth model template once the lips estimation procedure is completed 
may be denoted M* 

[0073] If energy £* is not initialized (258) or if energy E3 is less than E* (260), 
deformable mouth model template M* is set equal to deformable mouth model 
template M3 and E* is set equal to energy E3 (262). 
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[0074] Variable initl is incremented by one (264). 

[0075] If the value of variable initl is smaller than 3 (266), the method continues 
from block (242) at FIG. 9. Otherwise, variable initO is incremented by one (268). 
[0076] If the value of variable initO is smaller than 2 (270), the method continues 
from block (220) at FIG. 8. Otherwise, the method is terminated. 

ESTIMATING THE POSITION AND SHAPE OF THE TEETH 

[0077] The teeth positions are estimated in an iterative process using two different 

types of transformations. A "first-level teeth transformation" Tjt(YU, YD) is a fine 

transformation used to generally estimate the locations of contours CJTU and CJTD, 
to represent the difference in the opening of the jaw between the previously processed 
image and the current image. Briefly, FIG. 11 illustrates an exemplary first-level 
teeth transformation on a deformable mouth model template, with lip contours C_UU, 
C_UD, CJDU and CJDD omitted for clarity. The dashed contours 1100 and 1102 
belong to the template prior to the transformation, and the solid contours 1104 and 
1106 belong to the template after the transformation. YU is a relative vertical 
displacement in the C_TU contour, and YD is a relative vertical displacement in the 
CJTD contour, as summarized below in Table 4. In the first-level teeth 
transformation, the locations of the control points of contours C_TU and CJTD are 
transformed in unison, as shown in Table 5, which is explained in detail hereinbelow. 

[0078] A "zero-level teeth transformation" T 0T (YDISPL(i } k)) is a superfine 

transformation used to refine the estimated location of individual control points of the 
teeth contours C_TU and C_TD. Briefly, FIG. 12 illustrates an exemplary zero-level 
teeth transformation on a deformable mouth model template, with Up contours C_UU, 
C_UD, C_DU and C_DD omitted for clarity. The dashed contours 1200 and 1202 
belong to the template prior to the transformation, and the solid contours 1204 and 
1206 belong to the template after the transformation. Transformation parameters 
YDISPL(hk) represent relative vertical displacements between corresponding control 
points on teeth contour C_TU and C_TD. 

[0079] The first-level and zero-level teeth transformations are used several times and 

in a defined order to refine the estimates of the teeth positions. It should be 

understood that in alternative embodiments of the invention, a different number of 
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teeth transformation types may be used. Moreover, it should be understood that in 
alternative embodiments of the invention, teeth transformation types may be used a 
different number of times and in a different order or in no specific order. 
[0080] FIGS. 13 and 14 are a flowchart illustration of an exemplary method for 
estimating teeth positions in one of a sequence of images of a human face according 
to some embodiments of the invention. 

[0081] Referring to FIG. 13, a variable "initteeth", used as a loop counter, is 
initialized to the value zero (302) and deformable mouth model template Mfor the 
current image is initialized to be equal to deformable mouth model template M* 
(304). The value of variable initteeth is checked (306). 

[0082] If the value of variable initteeth equals zero, then the control points of 
"contour_teeth_up" (C_TU) and "contour_teeth_down" (C_TD) of deformable mouth 
model template M are initialized to coincide with the respective control points of 
"contour J:eeth_up" (C_TU) and "contour_teeth_down" (C_TD) of deformable mouth 

model template Mp of the previously processed image (308). 

[0083] If the value of variable initteeth equals 1, then "contour_teeth_up" (C_TU) is 
initialized to coincide with "contour_teeth_down" (CJTD) of deformable mouth 
model template M, and is then positioned vertically halfway between 
"contour_up_down" (CJJD) and "contour_down_up" (CJDU) (310). 
[0084] If the value of variable initteeth equals 2, then "contour_teeth_up" (C_TU) is 
initialized to coincide with "contottr_teethdown" (C_TD) and "contour_down_up" 
(C_DU) of deformable mouth model template M (312). 

[0085] Vertical edges map f " (x,y) is filtered using known or future methods, 
for example, the "MAX" filter of equation (8) with a WI6A window to generate a 
"filtered vertical edges map" (314). 

[0086] Transformation parameters YU and YD of a first-level teeth transformation 
TjxPTUtYD) are determined by miiiimizing a teeth energy objective function 

defined in equation (55) and calculated using blurred mouth images Tg, Ip> and I, the 

filtered spatial luminance peaks and valleys map and the filtered vertical edges map 
(316). 
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[0087] Deterrnining parameters YU and YD is demonstrated in the following equation 
and can be obtained using known or future methods, such as, for example, gradient 
descent, downhill-simplex, and the like. 

(16) YUJD = BigmmE T (T ]T (YU 9 m\l) 

[0088] A defonnable mouth model template M4 for the current image is generated 
(3 1 8) by transforming deformable mouth model template M using first-level teeth 
transformation T IT (YU t YD). 

[0089] Deformable mouth model template M is initialized with deformable mouth 
model template M4 (320). 

[0090] Transformation parameters YDISPL(i,k) of zero-level teeth transformation 
Tox(YDJSPL(i,k)J are determined by minimizing teeth energy objective function 

defined in equation (55) and calculated using mouth images 7#, 7p, and 7, the spatial 

luminance peaks and valleys map and the vertical edges map (322). 
[0091] Detemiining parameters YDISPL(i>k) is demonstrated in the following 
equation and can be obtained using known or future methods, such as, for example, 
gradient descent, downhill-simplex, and the like. 

(17) rozspz(a)^^ 

[0092] Referring now to FIG- 14, a deformable mouth model template M5 for the 
current image is generated (324) by transforming deformable mouth model template 
M using zero-level teeth transformation TojfYDISPLfrk)). X coordinates of the 

control points of deformable mouth model template Mj are calculated (326) using the 

following equations: 

(18) X 5 (z, P_C) = l A X(z, P JL) + l A X(z, P_R), i = P_LC, P_C, P_RC 

(19) X' (1, P_LC) = V* Xft P_L) + l A X(z, P JR.), i - P_LC, P_C, P_RC 

(20) X 5 (z, P_RC) - l A X(z, P_L) + 3 / 4 X(z, P_R), i = P_LC, P_C, P_RC 
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[0093] Energy £5, corresponding to deformable mouth model template M5, is 
calculated \ising contour and control point values of deformable mouth model 
template M5 in the equation for teeth energy objection function (328). 

[0094] The minimum energy of once the teeth estimation procedure is completed 
is denoted E**. In addition, the final deformable mouth model template once the 
teeth estimation procedure is completed is denoted M** 

[0095] If energy £** is not initialized (330) or if energy E5 is less than £** (332), a 
deformable mouth model template M** is set equal to deformable mouth model 
template M5, and £** is set equal to energy E5 (334). 
[0096] Variable initl is incremented by one (336). 

[0097] If initteeth is less than 3 (338) the method continues from block (304) in FIG. 
13. Otherwise, deformable mouth model template M** may be the resulting 
deformable mouth model template for the current image (340), and may be saved as 

Mp for calculations for the next image (342). 

TRANSFORMATION PARAMETERS 



[0098] The transformation parameters of the lip transformations and teeth 
transformations are summarized below in Table 4. 

Table 4 



Transformation 
Parameter 


Represents 


DX 


relative horizontal displacement of center of mouth 


DY 


relative vertical displacement of center of mouth 


DY11 


difference in location of upper lip relative to center of mouth 


DYd 


difference in location of lower lip relative to center of mouth 


DW 


difference in estimated mouth stretching 


YDISPL& k) 


relative vertical displacement of a particular control point 


XL 


relative horizontal displacement of control point P L 


YL 


relative vertical displacement of control point P L 


XR 


relative horizontal displacement of control point P R 


YR 


relative vertical displacement of control point P R 


YU 


relative vertical displacement of C TU contour 


YD 


relative vertical displacement of C TD contour 
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[0099] The effect of each transformation on the X-coordinates and Y-coordinates of 
the control points of the deformable mouth model template is summarized below in 
Table 5. For each transformation, the X- and Y-coordinates of a control point prior to 
the transformation are denoted X and Y 5 respectively, while the X- and Y-coordinate 
of the control point after the transformation are denoted X' and Y\ respectively. 



Table 5 



Transformation 


Effect on Mouth Model Template 


T 2L (DX,DY) 

second-level lip transformation 


Y' (z, k) = Y(z, k) + DY 

i = CUU,CUD,C DU, C DD 

/fc = P LC,P C,P RC 
X 5 (P L) = X(P L)+DX 
X'(P R) = X(P R) + DX 
Y 3 (P L) = Y(P L) + DY 
Y' (P_R) - Y(P_R) + DY 


T 1L (DX, DY, DYu, DYd, DW) 

first-level lip transformation 


Y' (z, A) = Y(z, A:) + DY + V* DYu 

/ = CUU,CUD 

k=P LC,P RC 
Y' 0', k) = Y(/", A:) + DY + V* DYd 
j = C DU,C DD 

&=P LC,P RC 
Y'(z, P_C) = Y(z, P_C) + Dr+ Dr« 

z = C UU, C UD 
Y'(i,P C) = Y(z,P C) + DY+DYd 

z = C DU,C DD 
X'(P L) = X(P IS) + DX-V 2 DW 
X'(P R) = X(P R) + DX+ Vi DW 
Y'(P L) = Y(P L) + DY 
Y'(P_R) = Y(P_R) + DY 


T- „ /Vf VJ? VT VP VTiTVPT ft Ir\\ 
1 0jr (AL, Alt, 11., M, lJJl&MrL,\l,Hj) 

zero-level lip transformation 


Y* (f, £) = Y(z, £) + YDISPL(i, k) 

z = C UU, C UD, C DU, C DD 
£ = P LC,P C,P RC 

X' (P L) = X(P L) 

X' (P R) = X(P R)+^3? 

Y' (P L) = Y(P L) + 7Z 

Y' (P R) = Y(P R) + 7if 


T 1T (YU, YD) 

first-level teeth transformation 


Y'(C TU, /c) = Y(C TU, k) + YU 
k = ? LC,P C,P RC 

Y'(C TD, k) = Y(C TD, k) + YD 
£=P_LC,P_C,P_RC 


T 0T (YDISPL(i,k)) 
zero-level teeth transformation 


Y' (t k) = Y(z, k) + YDISPLfl, k) 
z = C TU,C TD 
k=V LC,P C,P RC 
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THE LIPS ENERGY OBJECTIVE FUNCTION 

[00100] Lips energy objective function defined hereinbelow in equation 

(21), may be minimized to determine optimal transformation parameters for second- 
level lip transformation T 2 l(DX ) DY) (block -214- of FIG- 8), first-level lip 
transformation T 1L (DX, DY, DYu, DYd t DW) (block -236- of FIG. 9) and zero-level 
lip transformation T 0L (XL S XR, YL, YR, YDISPLp, k)) (block -252- of FIG. 9). 
[00101] Known or future optimization methods, such as, for example, gradient 

descent, downhill simplex, and the like, may be used to minimize lips energy 
objective function . Moreover, initial transformation points for the optimization 
method may be equal to zero. 

(21) E L (T, /(x, y)) = E L e* t (7\ l(x, y)) + E L t* {t) 

[00102] A generic transformation variable T appears explicitly in each energy 

term to indicate that the deformable mouth model templates used to evaluate an 
energy term are those after a transformation. Therefore, each energy term involves 
transformation parameters and the lips objective energy function may be minimized to 
determine optimal transformation parameters. 

[00103] Equation (21) contains an internal energy term B^inu described in 

equation (22), which defines a certain penalty for model deformation, and an external 
energy term B^ ex u described in equation (23), which defines how well the 
transformed lips model matches the lips of a real image. 

E L ft) = k L <*" te '' E center ft) + ^ ei ™ ticSpE elaslicSP ft) 
+ k L bound E% ound (J) + k l elasticE^ lastic (T ) 
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(23) 



(T, l(x, y)) = k L textU reB El nire (M B , I B {x, y),M, l{x, y)) 
+ k L textwecE^ (M p , I p (x, y), M, l(x, y)) 
-k^eE^iTjfcy)) 

+ k 1 corners E c(JnjeJ . s (T , l(x, y )) 

+ k L gap sE^ eethGaps (T,l(x, y)) 

— k L topography Ef Qpography (T , l(x, y)) 



COEFFICIENTS FOR INTERNAL ENERGY TERM AND 
EXTERNAL ENERGY TERM 

[00104] Equation (22) for internal energy term E^im includes coefficients 

^center, ^ elasticSP* ^ bound and ^elastic that may be defined for each Up 
transformation level as described in Table 6 below. Similarly, equation (23) for 
external energy term it ^ includes coefficients lit gaps* k corners* ^tedge* 

*t topography* ^textureB and k L textureC that may be defined for each lip transformation 
level as described in Table 7 below. Although the invention is not limited in this 
respect, these coefficients may be empirically defined, and may require tuning. 

Table 6 



Level 


zero 


first 


second 


Coefficient 








,L 

K center 


5 


5 


5 


,L 

K elasticSP 


0.02 


0 


0 


,L 

K bound 


5000 


5000 


0 


.L 

K elastic 


0.4 


0.4 


0 



21 



Attorney Docket No.: P-6214-US 
Assignee: Intel Corporation 



Table 7 



Level 


zero 






Coefficient 








_ 

K saps 


100 


100 


100 


k corners 


2 


2 


0 


k ecfee 


0.6 


0.6 


0.6 


k topography 


0.3 


0.3 


0.3 


k textureB 


0.3 


0.3 


0.3 


K textiweC 


0.3 


0.3 


0.3 



INTERNAL ENERGY TERM 

Center energy term center 
[00105] The center energy term ^center ma Y be selected to attract the center of 

the deformable mouth model template to an approximate central position of the mouth 
region of interest in the facial image. An exemplary center energy term B^ center is 
presented in the following equation. It will be appreciated that center energy term 
center ma Y be calculated using a variation of the following equation or 
alternatively, using other methods. 

(24) E center ( T ) = E centerX ( T ) + E centerY CO 
[00106] The X and Y components of the exemplary center energy term center 

are presented in the following equations: 

0, \D w \<W/6 
{T)A({\DA-WI6)\ 2 



(25) 



centerX 



\D w \>W/6 



W/6 J ' 

(26) D w =W P - R )+ X ( P - L y 2 -W/ 2 

0, \D H \<H/6 
{T)A(^D H \-Hl6f 



(27) 



F L 



H/6 



\D H \>H/6 
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(28) 



/U-H/2 



where in equations (25) and (26), "fP 5 denotes the mouth image width as determined 
in block -82- of FIG. 3, and in equations (27) and (28), "H" denotes the mouth image 
height as determined in block -82- of FIG. 3. In equation (28), the index i runs over 
all contours and the index j over the control points P_LC, P_C and P_RC. 

Elastic spline energy term E^elasticSP 
[00107] The elastic spline energy term B^elasticSP ma y be selected to cause the 

lip contours to be smooth at first approximation. An exemplary elastic spline energy 
term B^elasticSP is presented in the following equation. It will be appreciated that 

elastic spline energy term J^elasticSP mSL Y be calculated using a variation of the 
following equation or alternatively, using other methods. 

7o.75F(z,P_C)-0.1257(z 5 P_i?) > j n 
+ 0315Y{UP _L)-Y{i,P _LC) ; 



(29) 



7? L 

^elasticSP 



(T) = 



mw 



'0.757 (z, P _ C) + 0.3757 P _ J?)Y 
-0A25Y(i 9 P_L)-Y(i,P_RC) j 



i = C_UU, CJJD, C_DU, C_DD 
where "mw" denotes the width of the mouth in the base image. 

Bound energy term bound 
[00108] The bound term B^ bound ma Y be selected to ensure that the deformable 

mouth model template derived by minimizing lips energy objective function B^ 
represents a physiologically possible state of a mouth. An exemplary bound energy 
term bound is presented in the following equation. It will be appreciated that bound 
energy term ^ bound ma Y be calculated using a variation of the following equation or 
alternatively, using other methods. 
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E Lu»d (?) - S 
(30) " 



' PAR(Y(C _UD,p)-Y(C _UU,p),0,2UH)+ } 
PAR(Y(C _DU,p)- Y{C _ UU, p),0, 2W/3) + 

k par(y(c_dd, p )-y(c_du,p\o,2dh) 



+ ]TPAR(dy(c\dyO(c),dyl(c)) 

c 

where the function PAR is given in the following equation: 

r 0, ;te[jc0 5 ;d] 
(3 1) PAR(x 9 xO,xI) = < (x - jcO) 2 , x < xQ 

(x-xl) 2 , x>xl 

and dy, dyO and dyl are given in the following set of equations: 



(32) 


dy{c) = Y{c,P_C)-yl 




(33) 


yO = {Y(c,P_L) + Y(c,P_R))/2 




(34) 


yl = (Y{c,P_LC)+Y(c,P_RC))/2 




(35) 


y2 = (yl-y0)/4 




(36) 


dyO(C_UU) = y2- mw/16 




(37) 


dyO(C _ UD) = y2 - mw/8 




(38) 


dyO(C_DU) = min(max(y2 - mw/32,0\- 


-mw/16) 


(39) 


dyO(C_DD) = min(max(>>2 - mw/16,0),- 


■mw/8) 


(40) 


dyl(C_UU) = y2 + mw/16 




(41) 


dy\{C_UD) = y2 + mw/16 




(42) 


dyl(C_DU)=mm(y2 + mw/8,0) 




(43) 


dy\(C_DD) = min(max(y2 - mw/32,0),mw/S) 



and in equations (30) - (43), W denotes the width of the mouth in the base image, 
denotes the height of the lower lip in the base image, and "f/F denotes the 
height of the upper lip in the base image. 

Elastic energy term ^elastic 
[00109] The elastic energy term ^elastic ma Y be selected to describe two sets 

of virtual vertical springs connecting the inner and outer contours of the lips and to 
describe a virtual horizontal spring connecting the mouth corners. An exemplary 

elastic energy term ^elastic * s presented in the following equation. It will be 
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appreciated that elastic energy term ^elastic ma Y be calculated using a variation of 
the following equation or alternatively, using other methods. 



\2 

ww. - mw 



mw 



V. mw J J 



0.2 £ [f 64 yU " - yub " ) 2 

Pe(P LC.P CP RC)[\ mw J 



Pe(P_LC,P_C t P_RC) 

where "yup" denotes the height of the upper lip at the P control point in the current 
image, "ydp" denotes the height of the lower lip at the P control point in the current 
image, "yubp" denotes the height of the upper lip at the P control point in the base 
image, and "ydbp" denotes the height of the lower lip at the P control point in the 
base image. 

EXTERNAL ENERGY TERM 
Texture energy term 

texture 

[00110] An exemplary texture energy term ft texture is presented in the 

following equation. The texture energy term involves mouth images. When the lips 

objective energy is calculated during the iterations with second-level lip 

transformations, double-blurred mouth images are used in the texture energy term 

instead of unblurred mouth images. When the lips objective energy is calculated 

during the iterations with first-level lip transformations, blurred mouth images are 
used in the texture energy term instead of unblurred mouth images. It will be 

appreciated that texture energy term texture ma y be calculated using a variation of 

the following equation or alternatively, using other methods. 
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EL U re (M A , A{x, y), M B , B(x, y)) = 

-7-7 S^jI-^+^j+^F 



(45) 



R 

+ — 1— ^(^(x 1 >;)-5(x'(4>m(^>'))) 2 

^ \ A DL ) {x,y)czA DL 

where deformable mouth model template Mj$ and mouth image Bfey) are of a current 

image, and deformable mouth model template M4 and mouth image A fey) are of a 
previously processed or a base image; 
where S(Aj) is the area of Aj 9 

where dx^ and dyi are relative displacements between control point PJL of 
deformable mouth model template M^ and control point P_L of deformable mouth 
model template Mjj on two perpendicular axes "X" and "T\ respectively; and 
where dxR and dyj^ are relative displacements between control point P_R of 
deformable mouth model template Mj and control point P_R of deformable mouth 

model template Mb on two perpendicular axes "X s ' and "Y\ respectively. 

[00111] In equation (45), x'(x) and yufey) are coordinate transformation 

functions from control points of contours C_UU and C UD of deformable mouth 

model template to control points of contours C_UU and CMJD of deformable 

mouth model template Mb, and x'(x) and ydfey) are coordinate transformation 
functions from control points of contours C_DU and C_DD of deformable mouth 
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model template to control points of contours C_DU and CJDD of deformable 
mouth model template Mb, as detailed in the following equations: 

(46) x'(x) = (x- xL A )(xR B - xL B )/(xR A -xL A )+ xL B 
where parameters xLa and xRa are X coordinates of control points P_L and P_R, 

respectively, of deformable mouth model template Ma, and parameters xLq and xRg 
are X coordinates of control points P_L and P_R, respectively, of deformable mouth 
model template Mg; 
(47) 

ynj t r \ _ i v * (y ud B (*'(*)) - y uu B (*'(*))) 

(48) 

vd(x v W v - vdd (x\) * {ydd B {x'{x))-ydu B {x'{x))) 

y*{x,y)- [y ydd A { x ))* ^(^.^ W)+ ^(^)) 

where parameters yuu^fx), yud^(x), ydujtfx) and ydd^(x) are the Y coordinates of 
contour C_UU, C_UD, C_DU and C_DD S respectively, of deformable mouth model 
template M^ 5 and parameters yuugCx), yudsCx), ydu%(x) and yddgfx) are the Y 
coordinates of contour C_UU, CUD, C DU and C DD, respectively, of deformable 
moutli model template M#. 

Edge energy term edge 
[001 12] The edge energy term £^ edge ma Y be selected to attract the deformable 

mouth model template to vertical intensity gradients. An exemplary edge energy term 
^ edge is presented in the following equation, calculated as a sum of pixel values of 
the vertical edges map along contours C(C_UU), C(CJJD), C(C_DU), C(CJDU). 

m i Len(C(i)) ( XJ £c (0 

i = C_UU, C_UD, C_DU, C_DD 
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k; = 



\l, i = C_UU,C_DU 
[0.2, i = C_UD,C_DD 



where vertical edges map / " fay) is defined hereinabove in equation (4), "f 5 is 

an index of lip contours in the deformable mouth model template, C(i) represents the 

contour indexed by i 9 and Len(C(i)) denotes the length of contour C(z)- Coefficients kj 

may defined in relation to the sharpness of the lips when light fall on them from 
above. In an exemplary situation, when the edges of the upper lips are sharper than 
the edges of the lower lips, a coefficient may be defined larger for CJJU and C_DU 
than for CJJD and C_DD. 

[00113] When the lips objective energy E? is calculated during the iterations 

with second-level lip transformations, a double-filtered vertical edges map is used in 
the edge energy term instead of the unfiltered vertical edges map. When the lips 

objective energy E^ is calculated during the iterations with first-level lip 
transformations, a filtered vertical edges map is used in the edge energy term instead 
of the unfiltered vertical edges map. 

Corners energy term 

corners 

[001 14] The corner energy term ^comers ma y be selected to attract the corners 

of the lips to a specific edge structure. An exemplary corner energy term E? comers * s 
presented in the following equation. It will be appreciated that comer energy term 
^ comers ma Y b e calculated using a variation of following equation or alternatively, 
using other methods. 

EL,ers{T,l{x,y)) = 

(50 ) f {x, y ) + Y,^y)* lV - EDGE ^y) 

^(x,y)czA LC (x,y)cA RC > 

where S(Aj) is the area of Aj, and kernels KL and KR are defined as follows: 

1, x < X(P_L) ory = Y(P _L) 



(51) KL(x,y) = 



— 1 otherwise 
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(52) KR(x,y) = 



\l 9 x> X(P_R) or y = Y(P_R) 
- 1 otherwise 



[001 1 5] Graphically the kernels may be viewed as approximately 


+++++ 

+++++ 

+++++ 

for the right corner and as approximately 

+++++ 

+++++ 

+++++ 

+++++ 

for the left corner. The 5 sign is for -1 and the "+" sign is for +1 . 

[00116] When the lips objective energy is calculated during the iterations 

with second-level lip transformations, a double-filtered vertical edges map is used in 
the corners energy term instead of the unfiltered vertical edges map. When the lips 
objective energy is calculated during the iterations with first-level lip 

transformations, a filtered vertical edges map is used in the corners energy term 
instead of the unfiltered vertical edges map. 

Teeth Gap energy term E^TeethGap 
[00117] An exemplary teeth gap energy term E^TeethGap to describe the 

presence of a vertical tooth gap in the lip area is presented in the following equation. 
The teeth gap energy will be minimal to exclude teeth from the lip area. It will be 

appreciated that teeth gap energy term E^TeethGap ma Y be calculated using a variation 
of the following equation or alternatively, using other methods. 



(53) + 
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A l = A UL ~ ( A RC + A LC)> A 2 = A DL - ( A RC + 4td 
where S(Aj) is the area of A h and f°(x t y) is the teeth vertical gaps map defined 
hereinabove in equations (1) - (3). 

Topography energy term ^topography 
[001 18] An exemplary topography energy term ^topography t0 represent white 

teeth or a dark mouth cavity between the lips is presented in the following equation. 
It will be appreciated that topography energy term ^ topography ma y ^ e calculated 
using a variation of the following equation or alternatively, using other methods. 

(54) Ei„,, p „ r (Tj( X ,y))= 1 

where 5^4^) is the area of Aj, and f EAK VALLEY (x,y) is a spatial luminance peaks and 

valleys map, that can be obtained using known or future methods. 

[00119] When the lips objective energy is calculated during the iterations 

with second-level lip transformations, a double-filtered spatial luminance peaks and 
valleys map is used in the topography energy term instead of the unfiltered spatial 

luminance peaks and valleys map. When the lips objective energy is calculated 

during the iterations with first-level lip transformations, a filtered spatial luminance 
peaks and valleys map is used in the topography energy term instead of the unfiltered 
spatial luminance peaks and valleys map. 

THE TEETH ENERGY OBJECTIVE FUNCTION 

[00120] Teeth energy objective function defined herein below in equation 

(55) , may be minimized to determine optimal transformation parameters for first-level 
teeth transformation T 1T (YU t YD) (block -316- of FIG, 13) and zero-level teeth 

transformation T 0T (YDISPL(i } k)) (block -322- of FIG. 13). 

[00121] Known or future optimization methods, such as, for example, a 

downhill simplex method, may be used to minimize teeth energy objective function 
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Moreover, initial parameters for the downhill simplex method may be equal to 
transformation parameters that were computed for the previously processed image. 

(55 ) E T (Tj(x,y)) = E T ext (Tj(x,y))+ E T int {T) 

[00122] Equation (55) contains an internal energy term \ n u described in 

equation (56), which defines a certain penalty for model deformation, and an external 
T 

energy term E exU described in equation (57), which defines how well the 
transformed teeth model matches the teeth of a real image. 

(56) Ej nt (T) = E T bound (t)+ E T elastic (T)+k T eiasticSpE T elasticSp (T) 

El, (T, I(x, y)) = -k T ^E^, (T, T(x, y)) 

- k T ga P sE T TeethGaps (T, l(x,y)) 

(57) - k T intensity El ltensity (l , l(x, y)) 

+ k T dispersion Efc spersion (T ? l(x, y )) 

COEFFICIENTS FOR INTERNAL ENERGY TERM AND EXTERNAL 

ENERGY TERM 

T 

[00123] Equation (56) for internal energy term E f nt includes the coefficient 

T T 
k elasticSP* equation (58) below includes k ooim d, and equation (59) below includes 

T 

k spring- These coefficients may be defined for each teeth transformation level as 

described in Table 8 below. Similarly, equation (57) for external energy term E? ex t 

T T T T 

includes coefficients k e ^ g6f k gapS9 k intensity k dispersion that may be defined 

for each tooth transformation level as described in Table 9 below. Although the 
invention is not limited in this respect, these coefficients may be empirically defined, 
and may require toning. 
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Table 8 



Level 


zero 


111 a I 


Coefficient 






k bound 


10000 


10000 


K spring 


1 


2 


k elasticSP 


5000 


0 


Table 9 


Level 


Zero 


first 


Coefficient 






K edge 


2 


2 


v l 


10 


10 


k intensity 


1 


1 


k dispersion 


0 


0.25 



INTERNAL ENERGY TERM 

Bound energy term E? bound 

T 

[00124] The bound term E bound ma Y be selected to ensure that the deformable 

mouth model template derived by minimizing teeth energy objective function 

represents a physiologically possible state of a mouth. The teeth contours must not 

overlap and must be located between the inner lip contours. An exemplary bound 

T 

energy term E bound * s presented in the following equation. It will be appreciated 

that bound energy term ^ bound ma Y b e calculated using a variation of the following 

equation or alternatively, using other methods. 

(58) 

0, if Y(i D , j) > Y(i v , j) for all points j on all pairs of contours 
E T bomd {T) = \k\oun^m D J)-Y{i a J))\ if Y(i D3 j) < Y(i U9 j) 

iu= (C_UD, C_TU, C_TD} 
i D = {C_TU 3 CJTD, C_DU} 
j = {P_LC,P_C,P_RC} 
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Elastic energy term E elastic 
[00125] The elastic energy term E* elastic ma Y be selected to describe two sets 

of virtual vertical springs connecting the inner and outer contours of the lips with the 

teeth (control points of C JJU with C_TU and control points of C_DD with CJTD). 

T 

An exemplary elastic energy term E elastic is presented in the following equation. It 

will be appreciated that elastic energy term E 1 elastic ma Y be calculated using a 
variation of the following equation or alternatively, using other methods. 



(59) 



E elastic (^0 ~ {^spring ) ^ 



Y(C_UU,j)-Y(C_TU,j) 
\-Y B {C_ UU, j)+Y B {C_ TU, j\ 



2\ 



+ 



J 



r Y{C_TDj)-Y{C_DD,j) V 
^-Y B {C_TDJ)+Y B (C_DDJ\ 

j= {P_LC, P_C, P_RC} 
where Y(z, j) are the Y coordinates of control points in the current image, and Y^(i, j) 
are the Y coordinates of control points in the base image. 

Elastic spline energy term E^elasticSP 
[00126] The elastic spline energy term elasticSP ma y De selected to cause the 

teeth contours to be smooth at first approximation. An exemplary elastic spline 
energy term E* \lasticSP i s presented in the following equation. It will be appreciated 

that elastic spline energy term i^e/astfcSP ma Y be calculated using a variation of the 
following equation or alternatively, using other methods. 

'0.75 7(z , P _ C) - 0. 1 25 r(z, P_R)^ 
+ 0.375F(i,P _L)-Y(i,P _LC) J 
+ 

( 0.7 5Y(i,P _C)+ 0.37 5Y(i,P _R)^ 2 
0A25Y(i,P _L)-Y(i,P _RC) 



(60) 



'elasticSP 



(r)=Z 



J J 
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i = C_TU, CJTD 

EXTERNAL ENERGY TERM (82) 

T 

Edge energy term E e a ge 

[00127] An exemplary edge energy term ^ edge is presented in the following 

equation. It will be appreciated that edge energy term E^ edge mdL Y be calculated using 
a variation of the following equation or alternatively, using other methods. 

m e^j(x,y))= Yi"- EDG£ ^y)- E''-""^) 

( Xi y)&C_TD (x y y)eC_TU 

where the vertical edges map F' EDGE (x,y) is given in equation (4) hereinabove. 

Dispersion energy term E? dispersion 
[00128] The dispersion energy term ^dispersion is selected to correspond to 

image intensity dispersions in the teeth areas. An exemplary dispersion energy term 
E T dispersion is presented in the following equation. It will be appreciated that 
dispersion energy term I? dispersion ma Y be calculated using a variation of the 
following equation or alternatively, using other methods. 

( j V 

E T dispersion (T,i(x, y ))= x fayh-^m T^y) 



(62) 



f 1 V 



V ^ V M 2 ) {x % y )c:A 2 J 

A j ^'Axjt- ^RC + A LC ), A 2 = A DT - (A RC + A LC ) 

where S(Aj) is the area of A\. 

T 

Intensity energy term E intensity 
[00129] The intensity energy term ^intensity is selected to correspond to the 

average intensity (or peaks map) difference between the cavity of the mouth and the 
teeth areas. An exemplary intensity energy term I? intensity is presented in the 

following equation. It will be appreciated that intensity energy term ^intensity ma Y 
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be calculated using a variation of the following equation or alternatively, using other 
methods. 

Ktensity{ T A X >y)) = ^TJ\ Z 7 (*^) 

S \A)(x 9 y)czA 2 

— t~ — \ ^i{ x >y) 

S\^Mc)(x,y)c:A MC 

A] =A UT - (A R c + ^zc^ ^2 - ^z>r - (Arc + ^zc^ 

where 5(14^ is the area of ^4/, and JfftjiJ is an image intensity map or a spatial 
luminance peaks and valleys map, obtained using known or future methods. 

Teeth gap energy term TeethGap 
[00130] The teeth gap energy term E^TeethGap ma Y be selected to describe 

vertical teeth gaps or vertical edge is teeth areas (between contours C TJD and C_TU 
and between contours C_TD and C_DU) and their absence in the mouth cavity 
(between the C_TU and CJTD contours). An exemplary teeth gap energy term 

E> V TeethGap * s presented in the following equation. It will be appreciated that teeth 

gap energy term E^TeethGap ma Y be calculated using a variation of the following 
equation or alternatively, using other methods. 



(64) 



S \4- MC )(x,y)czA MC 



A j =A UT - (Arc + Aic), A 2 = A DT - (Arc + A LC ) 
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where S(Aj) is the area of A% % and F G (x M y) is the teeth vertical gaps map defined 
hereinabove in equations (1) - (3). 

[00131] FIG. 15 is a simplified block diagram of an exemplary apparatus 500 

to execute a method for estimating position and shape of lips and for optionally 
estimating position of teeth in one of a sequence of images of a human face according 
to some embodiments of the invention. Apparatus 500 may comprise a processing 
unit 502 and a memory 504 coupled to processing unit 502. 

[00132] A non-exhaustive list of examples for apparatus 500 includes a desktop 
personal computer, a work station, a server computer, a laptop computer,, a notebook 
computer, a hand-held computer, a personal digital assistant (PDA), a mobile 
telephone, a cellular phone, a video phone, a teleconference station, a video camera, a 
stills camera, a video game station, a surveillance equipment, and the like. 
[00133] A non-exhaustive list of examples for processing unit 502 includes any 
combination of the following: 
processors, such as 

a central processing unit (CPU), a graphics processing unit (GPU), a 
graphic accelerator, a digital signal processor (DSP), a reduced 
instruction set computer (RISC), a complex instruction set computer 
(CISC), and the like, 
and logic devices, such as 

complex programmable logic device (CPLD), a gate array, a field 
programmable gate array (FPGA), an application specific integrated 
circuit (ASIC), an application specific standard product (ASSP), and 
the like. 

[00134] Moreover, processing unit 502 may be implemented in part or in whole in an 
application specific integrated circuit (ASIC), an application specific standard product 
(ASSP), and the like. 

[00135] Memory 504 may be fixed in or removable from apparatuses 500. A non- 
exhaustive list of examples for memory 504 includes any combination of the 
following: 

semiconductor devices, such as 
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synchronous dynamic random access memory (SDRAM) devices, 
RAMBUS dynamic random access memory (RDRAM) devices, 
double data rate (DDR) memory devices, static random access memory 
(SRAM), flash memory devices, electrically erasable programmable 
read only memory devices (EEPROM), non-volatile random access 
memory devices (>TVTIAM), universal serial bus (USB) removable 
memory, and the like, 
optical devices, such as 

compact disk read only memory (CD ROM), and the like, 
and magnetic devices, such as 

a hard disk, a floppy disk, a magnetic tape, and the like. 
[00136] The method for estimating position and shape of lips and for optionally 
estimating position of teeth in at least one of a sequence of images of a human face 
according to some embodiments of the invention may be implemented in an 
executable or interpretable binary code 506 stored in memory 504. Alternatively, the 
method may be partly implemented in executable or interpretable binary code 506 and 
may be partly hardwired in processing unit 502. 

[00137] Memory 504 may include one or more image buffers 508 to store 

images, such as, for example, images of the sequence. 

[00 138] Some portions of the detailed description which follow are presented in 

terms of algorithms and symbolic representations of operations on data bits or binary 
digital signals within a computer memory. These algorithmic descriptions and 
representations may be the techniques used by those skilled in the data processing arts 
to convey the substance of their work to others skilled in the art. 

[00139] An algorithm is here, and generally, considered to be a self-consistent 

sequence of acts or operations leading to a desired result. These include physical 
manipulations of physical quantities. Usually, though not necessarily, these quantities 
take the form of electrical or magnetic signals capable of being stored, transferred, 
combined, compared, and otherwise manipulated. It has proven convenient at times, 
principally for reasons of common usage, to refer to these signals as bits, values, 
elements, symbols, characters, terms, numbers or the like. It should be understood, 

37 



Attorney Docket No.: P-6214-US 
Assignee: Intel Corporation 

however, that all of these and similar terms are to be associated with the appropriate 
physical quantities and are merely convenient labels applied to these quantities. 
[00140] Unless specifically stated otherwise, as apparent from the following 

discussions, it is appreciated that throughout the specification discussions utilizing 
terms such as "processing," "computing," "calculating," "determining," or the like, 
refer to the action and/or processes of a computer or computing system, or similar 
electronic computing device, that manipulate and/or transform data represented as 
physical, such as electronic, quantities within the computing system's registers and/or 
memories into other data similarly represented as physical quantities within the 
computing system's memories, registers or other such information storage, 
transmission or display devices. 

[00141] The processes and displays presented herein are not inherently related 

to any particular computer or other apparatus. Various general purpose systems may 
be used with programs in accordance with the teachings herein, or it may prove 
convenient to construct a more specialized apparatus to perform the desired method. 
The desired structure for a variety of these systems will appear from the description 
below. In addition, embodiments of the present invention are not described with 
reference to any particular programming language. It will be appreciated that a 
variety of programming languages may be used to implement the teachings of the 
invention as described herein. 

[00142] While certain features of the invention have been illustrated and 

described herein, many modifications, substitutions, changes, and equivalents will 
now occur to those of ordinary skill in the art. It is, therefore, to be understood that 
the appended claims are intended to cover all such modifications and changes as fall 
within the spirit of the invention. 
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